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Abstract : This study investigates response surface methods for computer 
experiments and discusses some of the approaches available. Orthogonal arrays 
constructed for computer experiments are studied and an example application to 
a technology selection and optimization study for a reusable launch vehicle is 
presented. 


I. Introduction 

Multidisciplinary design optimization (MDO) is an important step in the 
conceptual design and evaluation of launch vehicles since it has a significant 
impact on performance and lifecycle cost. The objective in MDO is to search the 
design space efficiently to determine the values of design variables that optimize 
performance characteristics subject to system constraints. In launch vehicle 
conceptual design, system performance can generally be determined by the use of 
computerized analysis tools available in many disciplines. However, these 
complex sizing and performance evaluation computer programs utilize iterative 
algorithms. In many cases, they are expensive and difficult to integrate and use 
directly for MDO. 

An alternative is to utilize response surface methodology to obtain mathematical 
models that approximate the functional relationships between performance 
characteristics and design variables. A common approach used in response 
surface model building is to utilize central composite designs (CCD) from the 
design of experiments literature (Box and Draper, 1987) to sample the design 
space efficiently. With this approach, design analyses (experiments) are 
performed at the statistically selected points specified by a CCD matrix. The 
resulting data is used to construct response surface approximation models using 
least squares regression analysis. These response surface equations are then used 
for MDO and for rapid sensitivity studies. 

However, like most experimental designs, CCD are designed with the physical 
experiments in mind where the dominant issue is the variance of measurements 



of the response (Sharifzadeh, Koehler, Owen and Shott, 1989). In an physical 
experiment, there is usually some variability in the output response with the 
experiment repeated with the same inputs. In contrast, the output of computer 
experiments is (in almost all cases) deterministic. Generally, there is no 
measurement error or no variability in analysis outputs. Therefore, experimental 
designs constructed to minimize variability of measurements may not be the best 
choice for computer experiments (Sacks, Welch, Mitchell and Wynn, 1989; Sacks, 
Schiller and Welch, 1989; Currin, Mitchell, Morris and Ylvisaker, 1991; Owen, 
1991, 1994). 

In this study response surface methods for computer experiments are investigated 
and some of the approaches available in the literature are disscussed. The focus is 
on response surface model building using orthogonal arrays designed for 
computer experiments. An example application to a parametric cost optimization 
study for a reusable launch vehicle is presented. 

II. Response Surface Model Building Using Central Composite Designs 

Response surface methods (RSM) can be utilized for MDO in cases where 
computerized design tool integration is difficult and design effort is costly. The 
first step in RSM, is to construct polynomial approximations to the functional 
relationships between design variables and performance characteristics (e.g. 
weight, cost) (Craig, 1978; Joyner and Sabatella, 1990; Stanley, et. al., 1993). In the 
next step, these parametric models are used for MDO and to determine variable 
sensitivities. A quadratic approximation model in the form given below (1) is 
commonly used since it can account for individual parameter effects, second- 
order curvature or non-linearity (square terms), and for two-parameter 
interactions (cross terms). 

Y = b 0 + Z bi xi + Z buxi 2 + Z Z tyj x^xj (1) 

In this model, the xj terms are the input variables that influence the response Y 
(the performance characteristic to be optimized); and b G , bi and bij are estimated 
least squares regression coefficients, based on the design and analysis data 
obtained by sampling the design space (or by conducting experiments). 

This second-order model (1) can be constructed efficiently by utilizing central 
composite designs (CCD) from design-of-experiments (DOE) literature (Myers, 
1971: Box and Draper, 1987: Khuri and Cornell, 1987; Cornell, 1990). CCD are 

first-order (2 n ) designs augmented by additional points to allow estimation of the 
coefficients of a second-order model (Cornell, 1990; Box and Draper, 1987). CCD 
enables the efficient construction of a second-order response surface model with 

significantly less effort than would be required by a full factorial study (3 n ). CCD 
have been successfully utilized in response surface model building and MDO in 



many aerospace design applications using computerized design analysis tools 
(Schnackel and Dovenmuehle, 1990; Lepsch, Stanley and Unal, 1995: Unal and 
Stanley, 1993: Unal, Lepsch, Engelund and Stanley, 1996: Venter, Haftka and 
Starnes, 1996). 

In some cases, however, RSM using CCD may not result in a good representation 
of the response surface as may be evidenced by poor predictions of the design 
analysis results. The reasons for this problem can be mainly due to; 

1) The response surface is more complex than can be represented by a 
second order approximation model given by equation (1), 

2) There are other influential design variables and interactions other than 
those currently under study, 

3) The sample design points (experiments) specified by a CCD may not be 
suitable in terms of selection of these specific points for experimentation 
with computerized design analysis tools. 

The third problem is directly related to the choice of specific experimental design 
points. In order to address this problem and to improve response surface model 
building using computer experiments, a study was conducted. A literature search 
on this subject is summarized in section ID. 

III. Response Surface Model Building Methods for Computer Experiments 

For computer experiments, commonly used in aerospace design, there is generally 
no measurement error or no variability in analysis outputs, given a specific set of 
inputs. On the other hand, in a physical experiment, there is some variability in 
the output response with the experiment repeated with the same inputs. In other 
words, the distinguishing characteristic of a computer experiment is that the 
output is deterministic (Sacks, Welch, Mitchell and Wynn, 1989). 

Even though the outputs from computer experiments are deterministic, the 
problem of selection of inputs at which to run a computer code is still an 
(statistical) experimental design problem. The quantification of uncertainty 
associated with predictions from fitted models is also a statistical problem (Sacks, 
Schiller and Welch, 1989). 

For computer experiments, it seems necessary to introduce randomness in order 
to gauge how much an estimate using a model may differ from the true value 
obtained from an experiment (Owen, 1991). There appear to be two main 
statistical approaches to computer experiments, one based on Bayesian statistics 
and a frequentist approach based on sampling techniques (Owen, 1991). 

The Bayesian approach models a computer code as if it were a realization of a 
stochastic process (Owen, 1991). Bayesian approach treats the bias, or the 



systematic departure of the response surface from a linear model, as the 
realization of a stationary random function (Owen, 1991). This model has exact 
predictions at the observed responses and predicts with increasing error variance 
as the prediction point move away from all design points (Owen, 1991). The 
approaches for design and analysis of computer experiments using Bayesian 
statistics are given by Sacks et al (1989), Chaloner and Verdinelli (1995), Currin, 
Mitchell, Morris and Ylvisaker (1991), and Welch et al, (1992) in some detail. 
Bayesian approach to experimental design appear to be a growing area of 
research. However, the application of Bayesian experimantal design methods in 
real design analysis and optimization problems have been limited partly due to 
the lack of user friendly software (Chaloner and Verdinelli, 1995). Further 
development appears to be needed before they can be applied to practical design 
optimization problems. 

The frequentist approach, surveyed by Owen (1991) on the other hand, introduces 
randomness by taking function values that are partially determined by pseudo- 
random number generators. Then this randomness is propagated through to 
randomness in the estimate (Owen, 1991). Owen, (1994) lists a set of randomized 
orthogonal arrays for computer experiments. The Statlib computer programs 
(http://lib.stat.cmu.edu/designs/) to generate these orthogonal arrays are also 
listed (Koehler and Owen, 1991). 

The use of these orthogonal arrays in practice for response surface model building 
would be similar to utilizing central composite designs, with a potential of 
improving model accuracy for computer experiments. Further work is needed to 
study the advantages and limitations of these arrays for approximation model 
building and MDO in launch vehicle design. In the following section, an example 
application to a technology selection and optimization study for a reusable launch 
vehicle is presented. 

IV. Example Application: Launch Vehicle Technology Selection Study 

The complete design of a launch vehicle is a multidisciplinary process in which 
aerodynamics, propulsion, weights and sizing, structures, performance, heating, 
controls, operations and cost must be addressed (Stanley et al, 1993). While it is 
essential that each of these disciplines be addressed at the conceptual design level, 
it is equally vital to be able to perform this multidisciplinary analysis and 
optimization efficiently such that the numerous design options may be evaluated 
and understood rapidly. 

Traditionally, the objective in a MDO study has been to search the design space to 
determine the values of design variables that optimize a performance 
characteristic (such as weight) subject to system constraints. However, research 
shows that up to 85 percent of the life cycle cost is committed during the early 
design phase (Fabrycky and Blanchard, 1991). Therefore, significant cost savings 



could be realized if designers were better able to evaluate their designs on a cost 
basis. 

This study focuses on rapid multidisciplinary analysis and evaluation-on-a-cost- 
basis for technology selection of a dual-fuel, rocket-powered, single-stage-to-orbit 
launch vehicle (SSV) (Unal, Braun, Moore and Lepsch, 1995). Different material 
and technology options, together with critical design variables, are studied to 
optimize design, development, test and evaluation (DDT&E) cost using 
orthogonal arrays for computer experiments. Calculus-based optimizers could 
not have been used in this case since material and technology options selection 
require the study of design variables that have discrete values. This study has the 
following steps: 

1. Identify the design variables to be studied and alternative levels 

In this study, design of a single-stage-to-orbit launch vehicle referred to 
determination of the appropriate component weights, sizes, and reference costs. 
To simplify the analysis such that the problem is tractable, several design 
disciplines were decoupled from the present analysis (Unal, Braun, Moore and 
Lepsch, 1995). An existing vehicle geometry, aerodynamics database, and internal 
packaging analysis were used (Engelund, Stanley, McMillin and Unal, 1993). 

Data from aerodynamics, structures, heating, and other subsystems were fixed or 
scaled appropriately. Furthermore, the ascent flight-path and propulsion system 
were fixed at a set of previously computed optimum values (Braun, Powell, 
Lepsch, Stanley and Kroo, 1995: Lepsch, Stanley and Unal, 1993). 

Seven major vehicle sections accounting for most of the empty weight were 
selected for optimization. These were the liquid hydrogen tank (Lh2/Tank), 
liquid oxygen tank (Lox/Tank), hydrocarbon fuel tank (Lhc/Tank), wing section 
structure (Wing), wing tip-fin section structure (Tip-fin), basic structure (Basic) 
and secondary structure (Second). Each of these variables were studied at three 
technology levels represented by three different materials. The material options 
were aluminum (Al), aluminum-lithium (Al-li) and composite material (Comp). 
The objective of this investigation was then to determine the best combination of 
material options for the seven major vehicle sections optimized for DDT&E cost 
and empty weight. 

2. Design the experiment and select an appropriate orthogonal array 

Owen, (1994) lists a set of orthogonal arrays for computer experiments. For this 
study, an orthogonal array that enables the study of seven variables was selected 
(http: / / lib.stat.cmu.edu/ designs/ owen.small). Using this orthogonal array 
(Addelman and Kempthorne, 1961), the seven design variables can be studied at 
three levels (values) by conducting 18 design experiments. A full factorial design 
where all possible variable/material combinations are studied would have 




required 2,187 (3 7 ) experiments. Variable interactions were assumed to be 
insignificant for this study. 

Table 1: Seven Variable Orthogonal Array (Owen, 1994) 



A 

B 

C 

D 

E 

F 

G 

1 

1 

1 

3 

2 

3 

1 

1 

2 

3 

2 

2 

1 

1 

3 

1 

3 

2 

3 

1 

3 

2 

2 

1 

4 

1 

2 

1 

1 

2 

1 

2 

5 

3 

3 

3 

3 

3 

3 

2 

6 

2 

1 

2 

2 

1 

2 

2 

7 

1 

3 

2 

1 

3 

2 

3 

8 

3 

1 

1 

3 

1 

1 

3 

9 

2 

2 

3 

2 

2 

3 

3 

10 

1 

3 

1 

2 

1 

3 

1 

11 

3 

1 

3 

1 

2 

2 

1 

12 

2 

2 

2 

3 

3 

1 

1 

13 

1 

1 

2 

3 

2 

3 

2 

14 

3 

2 

1 

2 

3 

2 

2 

15 

2 

3 

3 

1 

1 

1 

2 

16 

1 

2 

3 

3 

1 

2 

3 

17 

3 

3 

2 

2 

2 

1 

3 

18 

2 

1 

1 

1 

3 

3 

3 


3. Conduct the orthogonal array experiments 

The eighteen matrix experiments were conducted using a weights and sizing 
routine and parametric cost estimating relationships (CER). The Configuration 
Sizing program (CONSIZ) developed at NASA Langley Research Center, is used 
to size the vehicle and determine the component weights. Within CONSIZ, the 
vehicle is modeled as a collection of components representing structure, 
subsystem, and propulsion elements. 

For the cost analysis, a set of parametric CERs were developed (Moore, Braun and 
Powell, 1995). These equations model reference DDT&E costs specific to the 
reusable single-stage-to-orbit vehicle described earlier. The CERs include only 
those costs that are directly related to design variables contained in the study. 
Other cost elements, such as system level program management and 
programmatic costs, software costs, fees, etc., are not included. For the purpose of 
this study, these other cost elements are considered to be independent of the 
design trades over which the optimization occurs. 




The CERs estimate reference DDT&E costs as a function of technology readiness 
level and weight. For major structural elements, the technology readiness level is 
a function of material choice: aluminum, aluminum-lithium or composite. The 
material cost differences reflect the differences in raw material cost, tooling costs 
and fabrication complexities between these three materials. The NASA 
technology readiness scale was used as the technology indicator. 

The analysis results of the 18 experiments for DDT&E costs and vehicle dry 
weight (empty vehicle without propellants) are presented in Table 2. 


Table 2: Analysis Results 


a 

Lh2 

Lhc 

Lox 

Wing 

Tip-fin 

Basic 

Second 

DDT&E 
Cost (%) 

Dry 

Weight (lb) 

i 

1 

1 

3 

2 

3 

1 

1 

90.51 

281680 

2 

3 

2 

2 

1 

1 

3 

1 

84.79 

235518 

3 

2 

3 

1 

3 

2 

2 

1 

87.66 

248010 

4 

1 

2 

1 

1 

2 

1 

2 

91.27 

331372 

5 

3 

3 

3 

3 

3 

3 

2 

80.34 

182707 

6 

2 

1 

2 

2 

1 

2 

2 

81.66 

250502 

7 

1 

3 

2 

1 

3 

2 

3 

94.44 

260794 

8 

3 

1 

1 

3 

1 

1 

3 

83.14 

231724 

9 

2 

2 

3 

2 

2 

3 

3 

82.99 

195852 

10 

1 

3 

1 

2 

1 

3 

1 

87.89 

257717 

11 

3 

1 

3 

1 

2 

2 

1 

85.05 

248926 

12 

2 

2 

2 

3 

3 

1 

1 

89.77 

256285 

13 

1 

1 

2 

3 

2 

3 

2 

83.91 

219808 

14 

3 

2 

1 

2 

3 

2 

2 

85.19 

239077 

15 

2 

3 

3 

1 

1 

1 

2 

81.72 

273576 

16 

1 

2 

3 

3 

1 

2 

3 

84.11 

217207 

17 

3 

3 

2 

2 

2 

1 

3 

82.58 

231244 

18 

2 

1 

1 

1 

3 

3 

3 

92.94 

233486 


In Table 2, level 1 corresponds to material choice one (Al), level 2 corresponds to 
material choice two (Al-Li) and level 3 corresponds to material choice three 
(Comp). The cost values are normalizedand displayed as a percentage of the 
highest DDT&E cost vehicle. For the 18 material combinations shown in Table 2, 
the lowest cost is 80.34 % ( experiment number five). 









4. Analyze the data to determine the optimum levels and verify results 


The average cost (%) for each variable for each of the three levels are calculated 
and displayed in the response table given in Table 3. This response table shows 
the cost effects of the variables at each level. These are separate effects of each 
parameter and are commonly called main effects (Phadke, 1989). The average 
costs shown in the response table are calculated by taking the average for a 
variable at a given level, every time it was used. As an example, the variable Lh2 
was at level 2 in experiments 3, 6, 9, 12, 15 and 18. The average of corresponding 
costs is 86.12 (%) which is shown in the response table (Table 3) under Lh2 at level 
2. This procedure is repeated and the response table is completed for all variables 
at each level. 


Table 3: Response table 


Level 

Lh2 

Lhc 

Lox 

Wing 

Tip-fin 

Basic 

Second 

1 

88.69 

86.20 

88.02 

88.36 

83.91 

86.51 

87.61 

2 

86.12 

86.35 

86.19 

85.14 

85.58 

86.35 

84.02 

3 

83.52 

85.77 

84.12 

84.82 

88.72 

85.50 

86.70 


The optimum level for the design variables can now be selected by choosing the 
level with the lowest relative cost percentage. For example the lowest cost 
occured when variable Lh2 was at level 3 at 83.52 % as opposed to 88.69 % at level 
1, and 86.12 % at level 2. Similarly, the levels that optimize total DDT&E cost 
were chosen. The optimum levels are indicated by bold in Table 3. 

As the next step, least squares regression analysis is used to fit the second order 
approximation model (Equation 1) to the cost data (Yi) given in Table 2 in terms of 
the seven design variables (Xi). This parametric model accounts for the response 
surface curvature (square terms) and two factor interactions (cross terms). 

DDT&E Cost = 111.71 - 2.58 (lh2) + 1.22 (lhc) - 1.95 (Lox) - 7.61 (Wing) 

- 0.69 (Tip-fin) + 0.94 (Basic) -13.04 (Second) - 0.36 (lhc) 2 
+ 1.46 (Wing) 2 + 0.79 (Tip-fin) 2 " 0.36 (Basic) 2 
+ 3.15 (Second) 2 


Note that, in this response surface approximation model, the parameter values are 
restricted to 1 (Al), or 2 (Al-Li), or 3 (composite). 




Table 4: Minimum cost value and corresponding material choices 


Cost 

Weight (lb) 

76.29 

185275 


Lh2/Tank 

Lhc/Tank 

Lox/Tank 

Wing 

Tip-fin 

Basic 

Second 

Comp 

Comp 

Comp 

Comp 

Al 

Comp 

Al-li 


At these levels, the DDT&E cost was predicted to be 75.02 % using a second order 
prediction model. As a next step, a verification analysis was performed. The 
weight and cost of a vehicle constructed from these material choices were 
computed to be 185,275 lb. and 76.29 % recpectively (Table 4). 

V. Conclusions 

This study presents a brief overview of the response surface methods (RSM) for 
computer experiments available in the literature. The Bayesian approach and 
orthogonal arrays constructed for computer experiments (OACE) were briefly 
discussed. An example application of OACE to a cost optimization study for a 
launch vehicle was also given. In this case study, an orthogonal array for 
computer experiments was utilized to build a second order response surface 
model. Gradient-based optimization algorithms could not be utilized in this case 
study since the design variables were discrete valued. 

Using OACE, optimum combination of material choices for seven launch vehicle 
technologies that minimize DDT&E cost were determined. Similar results were 
obtained in a previous study using a three level fractional factorial experimental 
design (Unal, Braun, Moore and Lepsch, 1996). Specifically, the fractional 
factorial design used in the prior study was a Taguchi (L18) orthogonal array 
(Taguchi and Konishi, 1987). 

The use of OACE in this case for RSM did not show any advantage over the use of 
three level orthogonal arrays. The results obtained were the same in both 
methods. This perhaps could be expected in this case since the variables studied 
could only take three discrete values and variable interactions were assumed to be 
insignificant. However, the OACE listed by Owen (1991) seem to offer an efficient 
alternative to Taguchi's multilevel (three or more levels) orthogonal arrays 
(Taguchi and Konishi, 1987) for MDO cases that require the study of discrete- 
valued variables. Further work is needed in applying OACE to test cases with 
continuous variables to determine the advantages and limitations of OACE. 
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